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ABSTRACT 



Along with assessing the value of a program, evaluations 
should aid in decision making, support change management, and improve 
processes. For an evaluation to effect change, three conditions should occur: 
informing, collaborating, and mixing. The evaluation of a Wheaton College 
(Massachusetts) graduation requirement that students complete a minimum of at 
least one out-of - class learning experience provides an example of an 
evaluation that served as a catalyst for change. The requirement moved from a 
five-step prescribed, structured reflection process to a three-step 
student- centered reflection process integrated into the academic community. 

As the example from Wheaton College shows, for an evaluation to have 
effective use, it has to be a shared, collaborative educational venture 
between the evaluator and those involved in the program. (Contains 16 
references.) (SLD) 
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Evaluation as a Catalyst for Change 

A.A.H.E. Assessment Conference 2001 

Evaluators want the findings and recommendations from their studies to be helpful to those 
who commissioned the evaluation and to those who are affected by it, Along with assessing the 
value and worth of a program, evaluations should aid decision-making, support change 
management, and improve processes, Many evaluators feel that they also have a responsibility 
to help others understand evaluation approaches and procedures, Each of these is a component 
of evaluation use. 

At Wheaton College in Norton, Massachusetts, an evaluation of a graduation requirement 
that students complete a minimum of one out-of-class learning experience (e.g„ campus 
leadership, community service, internship, on- or off-campus job, practicum) and a required 
reflection essay served as a catalyst for change, Evaluation data were and continue to be used 
to refine and assess the requirement, The requirement moved from a five-step, prescribed 
structured reflection process to a three-step, student-centered reflection process integrated into 
the academic community, Wheaton serves as example of evaluation structures that facilitate 
evaluation use. These are described below and may be applied to other settings. 

Evaluation use is a multidimensional concept that has been studied considerably over the 
last thirty years (Caracelli & Preskill, 2000), Although evaluation utilization is commonly thought of 
in terms of influence of results or evaluation data, it also includes other facets, Traditionally, 
evaluation utilization is thought to have four components: instrumental, conceptual, process, and 
symbolic (Johnson, 1998), Instrumental is results-based use. Conceptual use refers to the 
development of understanding, opinions, and attitudes towards a program and/or evaluation in 
general based on experiences with past evaluations. When behavior changes accompany 
cognitive understanding, process use occurs, For instance, one begins "to think like an evaluator, " 
Symbolic use occurs when results are used for political self-interest, such as to legitimize a preset 
opinion or earlier decisions, and can be problematic. Recently, Kirkhart (2000) conceptualized use 
in terms of a three-dimensional theory of influence (source of influence, intention, time frame), 

For an evaluation to affect change or to move from the file drawer to active use, three 
conditions should occur, These are informing, collaborating, and mixing. Informing participants 
about evaluation thinking, approaches, and procedures, collaborating with users during all stages 
of an evaluation, and mixing evaluation approaches and methods should be inherent to an 
evaluation, Although they are discussed separately, the three conditions are not discrete, Rather 
than follow sequentially like hop-scotch, they over lap and move back and forth. 

Informing participants or potential users does not imply a didactic teaching approach, but 
using informal and formal mechanisms to learn what potential users already know and feel about 
evaluation and to guide them towards an understanding of evaluation methods, approaches, 
and use in their and other settings. For instance, at Wheaton's Filene Center where students may 



go to facilitate completion of their graduation requirement, staff are interested in self improvement 
and requested an evaluation workshop soon after the evaluation began. This request made 
informing easy. During a workshop, staff participated in exercises adapted from Michael Q. Patton 
(1 997). One of these exercises involved free association with the words, evaluation and evaluate, 
which revealed fears and perceptions. Metaphors for evaluation helped everyone understand 
how each other viewed evaluation. At a later point, some faculty and upper-level administration 
went through a similar exercise. These early exercises relieved a lot of tension and helped the 
evaluator make sure that concerns would be addressed. The activities also provided an 
opportunity for the evaluator to share her viewpoints and expertise on evaluation. On-going 
information sharing (staff to evaluator/evaluator to staff) since the first workshop is less formal. 

Informing can occur during workshops and at less formal times. Because many in higher 
education have experience with experimental research, the differences between evaluation and 
research, how evaluation is cyclical, and multiple purposes of evaluation are topics for discussion. 
Wheaton college and the program staff already have a tendency towards reflection and self- 
evaluation; integrating evaluation into day-to-day operations takes on a strengthening rather than 
introductory approach, For instance, evaluation's systematic, representative, and concrete 
approach to gathering data supplement anecdotal observations. 

Collaboration should start at the beginning of an evaluation and continue throughout. 
While the amount of participation depends on one's role in the program or evaluation, 
communication with everyone is essential. Determining what to evaluate, what questions to 
answer, and how to evaluate is a joint effort with key people. It is also an opportunity to instill 
evaluation “thinking." Focusing, planning, and clarifying together help facilitate understanding 
and acceptance of an evaluation. The following questions that were posed at the start of the 
Wheaton's evaluation can be asked for any evaluation: 

• what is the program to be evaluated - its goals, objectives, activities, resources, its 
context? 

• what will be evaluated? 

• what are the purposes of the evaluation? 

• who wants and needs the evaluation? 

• who will use the results? 

• how will the results be used? 

• what types of decisions will be made? 

• what questions need to be answered or issues should be addressed? 

• what type of information is needed? and whether it is feasible to gather that type 
of information 

• what type of information will be accepted? e,g„ anecdotes, statistics 

• is there really enough programmatic emphasis to have an effect? where is the 
greatest emphasis? 



After answering the questions, develop and prioritize a series of evaluation questions and 
select related methods of data collection. Some people find an evaluation timeline forevaluation 
questions and related data collection helpful. Reporting of findings from data collection is shared, 
frequent, and timely. 

When the evaluation emphasis is on what is most appropriate to conducting a good 
evaluation or what best answers evaluation questions, a mix of methods and approaches is 
suitable. Influenced by utilization-focused (Patton, 1 997), objectives-based, pragmatic, and quasi- 
experimental approaches, Wheaton's evaluation is an eclectic mix. The Standards for Evaluation 
(Joint Committee, 1994) guide the evaluation which is both formative and summative in nature. 
Qualitative (e.g„ open-ended interviews) and quantitative (e.g„ published surveys)methods are 
used. For instance, open-ended interviews and focus groups about the value of documentation 
of out-of-class experiences, lead to structured questions for members of the senior class about the 
documentation of their experiences. 

Example Caveat . As a result of evaluation data changes were made to the 
documentation process. These changes are now part of the evaluation. While informing, 
collaborating, and mixing played a part this change, without the support of the Wheaton 
administration and a culture of self evaluation, the change would be less smooth. For many 
involved, the findings support what they find and/or wish would happen. Receptiveness to 
evaluation in general is a key to use. 

For an evaluation to have effective use, it has to be a shared, collaborative, educational 
venture between the evaluator and those involved in the program. Using approaches and 
methods that are appropriate to evaluation questions rather than adherence to a favorite 
pedagogy can provide comprehensive understanding of how a program works and what its 
effects are. 
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